Content

Test Utility

What is Test Utility

Test Utility literally refers to using a test as a utility, as a tool. And in our context we mean using tests as a toal for selection.

Costs and benefits

  • monetary
  • non-monetary

The psychometric properties of the test determine the relationship between costs and benefits.

These properties can be empirically tested. We will show you the reasoning process in this lecture.

Predictive / Criterion Validity

Often we want to predict based on test scores how an individual will perform at a later time.

  • The thing we want to predict is called the criterion
    • Example: good employees or good students
  • The test we use to predict is obviously called the predictor

The predictive validity is defined as the correlation between the predictor / test score and the criterion.

NOTE: This is no different than the \(R\) in regular regression analysis; the correlation between the outcome and the model.

With a correlation of zero,
selection is useless

Cut-off scores

Based on our criterion and predictor we need to determine:

  1. When we trully know someone is competent.
  2. At what predictor score we want to select someone.

That means setting a:

  • Cut-off for the criterion (Base Rate)
  • Cut-off for the predictor (Selection Ratio)

Utility (interactive)

Counts and margins

FN TP 0
TN FP 0
0 0 0

What is the:

  • Correlation = 0.796
  • Hit Rate
  • Base Rate
  • Sensitivity
  • Specificity

CASE STUDY:

Admission test (NL: selectie aan de poort)

  • Test week data from UvA Psychology students
    • Motivation
    • Study results
  • The data was gathered before we actually had admission tests (all students where admitted).
  • Because of this, we can determine the effectiveness of a supposed admissions procedure.
  • We can calculate how much better our results would be if we had actually selected.
  • This comparison will show us how much there is to gain and the price we have to pay

  • Average Motivation: 0.417
  • Pass mark: 6
  • Correlation: 0.407

Hit Rate

FN
102 		TP
162
TN
163 		FP
85

Percentage of correct conclusions.

\[\definecolor{green}{RGB}{0,128,0} \text{Hit Rate} = \frac{{\color{green}\text{TP}} + \color{green}\text{TN}}{N}\]



\[\frac{162 + 163}{512} = 0.63\]

Base Rate

FN
102 		TP
162
TN
163 		FP
85

Percentage of correct conclusions when not selecting.

\[\definecolor{red}{RGB}{255,0,0} \text{Base Rate} = \frac{{\color{green}\text{TP}} + \color{red}\text{FN}}{N}\]



\[\frac{162 + 102}{512} = 0.52\]

Hit Rate - Base Rate

To determine the benefit of selection, we need to know what the hit rate is compared to the base rate.

\[\text{Hit Rate} - \text{Base Rate} = \text{Benefit}\]



\[0.63 - 0.52 = 0.11\]

Sensitivity

FN
102 		TP
162
TN
163 		FP
85

We can determine the efficiency of selection by calculating the sensitivity and specificity of a test.

Sensitivity: Percentage of eligible candidates that will be selected.

\[\text{Sensitivity} = \frac{\color{green}\text{TP}}{{\color{red}\text{FN}} + \color{green}\text{TP}}\]



\[\frac{162}{102 + 162} = 0.61\]

The lower the sensitivity
The more frustrated
parents and students

Specificity

FN
102 		TP
162
TN
163 		FP
85

Specificity: Percentage of inapt candidates that will be rejected.

\[\text{Specificity} = \frac{\color{green}\text{TN}}{{\color{green}\text{TN}} + \color{red}\text{FP}}\]



\[\frac{163}{163 + 85} = 0.66\]

Steps to determine efficiency

  1. Select predictor and criterion
  2. Calculate the correlation in an unselected sample
  3. Determine the efficiency based on possible cut-off scores

Only if you select everyone
you can determine
the quality of the procedure

Taylor-Russell Tables

So, what if you do not have the resources to conduct empirical research?

In that case, you can resort to using Taylor-Russell tables, hhich provide estimates of the percentage of eligible candidates of those selected.

  • TABLE: Base Rate
  • Y-AXIS: Predictive validity
  • X-AXIS: Selection ratio
  • CELLS: Proportion satisfactory among those selected = \(\frac{\color{green}\text{TP}}{{\color{green}\text{TP}} + \color{red}\text{FP}}\)

Influential factors

The two factors influencing the cell values “The proportion satisfactory among those selected” are:

  • Predictive validity
  • Selection ratio

Increase efficiency

So, there are two ways to increase efficiency

  • Increase predictive validity
  • Use more extreme selection ratio

Threats to validity

Criterion validity is threatened by all factors that reduce the relation between predictor and criterion.

  • Unreliable tests: noise reduces the relation.
  • Other validity issues
  • Wrong predictor
  • Restriction of range: restrictions in test scores results in lower correlation.

Restriction of range

Increase validity

Criterion validity can be increased by all measures that increase the correlation.

  • Higher reliability: more items, deleting bad items
  • Use better predictor
  • Less restriction of range

Just wanting selection
to work does not
influence the quality
of selection

Problems with selection in NL

  • Correlation between predictors and success is low
  • We have a bad selection ratio

Why selection works in US

  • High correlation
    • More variation vs NL
    • NL has restriction of range (VWO/HAVO/VMBO) vs high school in US
  • Prestigious Ivy League universities have good selection ratio
    • They select very few students; Harvard selects 10% highest SAT

Wrap-up

  • Good selection needs high criterion validity
  • The higher the correlation the higher the gain
  • The efficiency can be determined in advance.
    • When selection is implemented, you can not determine the efficiency
  • If criterion validity suffers from restriction of range, then the efficiency argument does not hold.
    • Policy should not be based on psychometric argumentation.

End